Generalizing from Case studies: A Case Study

Most empirical evaluations of machine learning algorithms are case studies { evaluations of multiple algorithms on multiple databases. Authors of case studies implicitly or explicitly hypothesize that the pattern of their results, which often suggests that one algorithm performs signiicantly better than others, is not limited to the small number of databases investigated, but instead holds for some general class of learning problems. However, these hypotheses are rarely supported with additional evidence, which leaves them suspect. This paper describes an empirical method for generalizing results from case studies and an example application. This method yields rules describing when some algorithms signiicantly outperform others on some dependent measures. Advantages for generalizing from case studies and limitations of this particular approach are also described. A central objective in machine learning research is to determine the conditions describing when one heuristic learning algorithmoutperforms others for a given set of dependent variables (e.g., predictive accuracy, speed, storage, etc.). Although formal mathematical analyses are preferred to detail these conditions in the form of average expected computational behavior (Pazzani & Sarrett, 1990), such results are diicult to produce since the algorithms and/or databases are usually complex. Instead, empirical evaluations are conducted to yield case study results { measures of some dependent variable(s) obtained from applying a set of algorithms to one or more carefully selected databases (e. Invariably, some algorithms are reported to signiicantly outperform others in the case study. Although authors usually hypothesize why these performance diierences occurred, their explanations are infrequently evaluated and may be inaccurate. More systematic methods are required to accurately generalize case study results. Few methods for generalizing case studies have been reported in the machine learning literature. However, the approach introduced in this paper has much in common with Rendell and Cho's (1990) investigations. They used artiicially-generated databases to examine how the performances of two similar algorithms were aaected by several data characteristics, particularly concept size (i.e., the percentage of positive instances) and concept concentration (i.e., the number of prototypes deening the target concept). This paper instead focuses on a general method that characterizes the situations when arbitrarily diierent learning algorithms have a constant signiicant performance diierence. More speciically, this paper details a simple empirical method that generalizes case studies. It is independent of the set of dependent and independent variables being investigated, the selected learning task, and the selected learning algorithms. The objective of this generalization method is to …